Skip to content

[executorch][llm] Load GGUF NVFP4 weights as ExportableNVFP4Tensor - #25338

Open
Gasoonjia wants to merge 5 commits into
gh/gasoonjia/213/basefrom
gh/gasoonjia/213/head
Open

Gasoonjia wants to merge 5 commits into
gh/gasoonjia/213/basefrom
gh/gasoonjia/213/head

Conversation

@Gasoonjia

@Gasoonjia Gasoonjia commented Nov 20, 2026 •

Copy link
Copy Markdown
Contributor

Stack from ghstack (oldest at bottom):

ExportableGGUFTensor now accepts GGML NVFP4 (type 40) blocks: 64 elements per block as four K16 sub-blocks, four UE4M3 scale bytes followed by 32 bytes of E2M1 codes. dequantize_gguf decodes them directly, and to_exportable_nvfp4_tensor() repacks them losslessly into the portable torchao NVFP4 layout (nibble reorder from GGML's eight-lows-then-eight-highs to even-element-low, GGML's sign-ignoring UE4M3 canonicalized to float8_e4m3fn, code 8 canonicalized to +0, optional per-tensor scale from the GGUF .scale companion). The 0xff scale byte decodes to 480 in GGML, which E4M3 cannot represent, so the conversion rejects it instead of silently changing values. ExportableNVFP4Tensor carries the optional activation per-tensor scale the CUDA kernels use.

Backends only need to understand ExportableNVFP4Tensor; none of them has to parse GGML blocks.

This is the first of three diffs adding NVFP4 to the CUDA backend (GGUF loading, CUDA kernels and dispatch, Muse Glimmer integration), followed by kernel follow-ups.

Authored with AI assistance.

Differential Revision: D131418678

[ghstack-poisoned]
@pytorch-bot

pytorch-bot Bot commented Nov 20, 2026 •

Copy link
Copy Markdown

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/25338

Note: Links to docs will display an error until the docs builds have been completed.

⏳ No Failures, 3 Pending

As of commit ca5ea65 with merge base 9f49fe2 (image):
💚 Looks good so far! There are no failures yet. 💚

This comment was automatically generated by Dr. CI and updates every 15 minutes.

@github-actions

Copy link
Copy Markdown

This PR needs a release notes: label

If your change should be included in the release notes (i.e. would users of this library care about this change?), please use a label starting with release notes:. This helps us keep track and include your important work in the next release notes.

To add a label, you can comment to pytorchbot, for example
@pytorchbot label "release notes: none"

For more information, see
https://github.com/pytorch/pytorch/wiki/PyTorch-AutoLabel-Bot#why-categorize-for-release-notes-and-how-does-it-work.

@Gasoonjia

Copy link
Copy Markdown
Contributor Author

@claude /pr-review

@claude

claude Bot commented Nov 21, 2026 •

Copy link
Copy Markdown

Claude finished @Gasoonjia's task in 21m 26s —— View job


Requesting changes: nothing owns the NVFP4 per-tensor .scale companion, act_per_tensor_scale is added with no reader and no defined meaning, and "nvfp4" in dequantize_gguf now reaches the native backend, which only fails on it when the model loads on device.

Detailed review

PR Review: #25338

Summary

This PR teaches ExportableGGUFTensor to read GGML NVFP4 (type 40) blocks, decode them in dequantize_gguf, and repack them into ExportableNVFP4Tensor, which gets a new act_per_tensor_scale attribute. Three design gaps need resolving before landing:

  • Per-tensor scale: nothing stores or applies the scale that GGUF keeps in the separate .scale tensor.
  • act_per_tensor_scale: the new attribute has no reader and no defined meaning.
  • Native backend: it accepts NVFP4 graphs that it cannot run.

The test gating and the 0xff reference coverage are also weaker than the test comments claim.

API Design

  1. Nothing stores or applies the NVFP4 per-tensor scale (extension/llm/export/gguf.py:418-422, :460-464, :646-678)

    • NVFP4 is the first GGUF type whose blocks are not self-contained: the description puts the per-tensor scale in a separate .scale tensor. But:
      • iter_gguf returns that .scale tensor as an unrelated plain F32 tensor.
      • ExportableGGUFTensor has nowhere to keep it.
      • dequantize() returns values without it, so the aten.linear / aten.embedding / _to_copy implementations (:613-640) do too.
      • to_exportable_nvfp4_tensor() silently defaults the scale to 1.0.
    • The module docstring's default path (keep the wrapper as a weight, then dequantize_gguf, then linear) therefore gives weights that are off by the per-tensor scale, with no error.
    • Loaders here skip tensor names they don't map (for example examples/models/muse-glimmer/loaders/checkpoint_loader.py:592-598), so they drop the .scale tensor without notice.
    • Fix: store the scale on the NVFP4 ExportableGGUFTensor (pair it in iter_gguf, apply it in dequantize_gguf / to_exportable_nvfp4_tensor). At minimum, document on iter_gguf, the class, and to_exportable_nvfp4_tensor that NVFP4 values exclude the per-tensor scale and that the caller must pair the .scale tensor by name.
  2. "nvfp4" in dequantize_gguf reaches the native backend, which fails only at device load (gguf.py:69)

    • This contradicts the description's claim that no backend has to parse GGML blocks. The chain:
      • Before this PR, iter_gguf rejected type 40 at load time.
      • Now backends/native/examples/hf_llm/export_hf_llm.py:147-160 keeps the weight as an ExportableGGUFTensor.
      • NativeSupportedOperators claims every dequantize_gguf node (backends/native/partitioner.py:49).
      • _fold_gguf_dequant writes codec gguf:nvfp4 (backends/native/serialization/graph_serialize.py:1052).
      • The ggml engine's kquant_type throws on anything but gguf:q4_k, gguf:q5_k and gguf:q6_k (backends/native/runtime/engine/ggml/GgmlTypes.cpp:23-35).
    • Export succeeds and the error moves to runtime.
    • Fix: claim or fold only the dequantize_gguf types the engines implement, so the error stays at export time. Fix this →
  3. act_per_tensor_scale is a new contract between the loader and the CUDA kernels, with no reader and no definition (extension/llm/export/nvfp4.py:76, :85, :97, :130; gguf.py:428, :473)

    • Nothing in the repo reads it: a grep finds only these assignments and pass-throughs.
    • dequantize() (nvfp4.py:100-108) does not pass it to torchao::dequantize_nvfp4, and aten.linear (:139-145) ignores it. Eager and exported graphs are therefore weight-only. The CUDA kernels can only receive the value by reading the subclass object during tracing, which is an undocumented side channel.
    • Its meaning is not written down: is it the ModelOpt input_scale or its reciprocal, and should eager linear quantize activations with it?
    • It is typed as a bare Python float, while per_tensor_scale is a validated 0-d tensor. Upstream torchao's NVFP4Tensor keeps act_per_tensor_scale as optional tensor data.
    • No test covers it, not even whether it survives .to().
    • Fix: move it to the stack PR that uses it, or document the contract on ExportableNVFP4Tensor and add a test.
  4. to_exportable_nvfp4_tensor breaks the conventions of the sibling conversion methods (gguf.py:424-429, :443-444)

    • Argument order: per_tensor_scale is the first positional parameter. to_exportable_int4_tensor takes output_dtype first, and callers pass it positionally (examples/models/gemma4_31b/cuda_packers.py:80). So w.to_exportable_nvfp4_tensor(torch.bfloat16) binds the dtype to the scale. Make the parameters keyword-only.
    • Error type: a wrong quant type raises ValueError. The siblings raise NotImplementedError("... only supports ...") (:487-490, :602-605).
    • Fix this →

Testing

  1. The gguf>=0.19 pin the tests rely on doesn't exist (extension/llm/export/test/test_gguf_nvfp4.py:21-26)

    • The comment says "OSS pins gguf>=0.19". In fact:
      • requirements-dev.txt:23 is a bare gguf.
      • The workflows install gguf with no version (test-native.yml:126, mlx.yml:76, cuda.yml:654,712).
      • extension/gguf_util/install_requirements.sh:8 pins gguf==0.6.0.
    • Any environment with gguf < 0.19 silently skips 3 of the 7 tests: every comparison against the gguf decoder and the only iter_gguf test. Also check which version fbsource//third-party/pypi/gguf provides to the new BUCK target.
    • The per_tensor_scale and .to(bfloat16) checks (:89-97) don't need gguf, since they could compare against w.dequantize(), but they are skipped too.
    • Fix: pin gguf>=0.19 (or correct the comment), and move those two checks into a test that always runs.
  2. The rule that scale byte 0xff decodes to 480 is never compared with the gguf decoder (test_gguf_nvfp4.py:66-68, :83, :103; gguf.py:101-102)

    • arange(255) stops at 0xfe, even though the comment says "Every scale bit pattern". The random tests clamp scale bytes to 254.
    • So the hand-coded 480 case, which is the stated reason for rejecting 0xff, is only checked against the hand-computed 240 at :143.
    • Fix: include 0xff when comparing w.dequantize() with reference(raw), and leave it out only of the conversion check.
    • Fix this → (covers items 5 and 6)
  3. No export test.

    • test_gguf.py:318 (test_linear_exports_with_dequantize_gguf) runs torch.export for the k-quants. Nothing exports NVFP4 weights in either form.
    • So two things are unchecked: which nodes the graph contains (dequantize_gguf(…, "nvfp4") or dequantize_nvfp4), and whether the subclass with the new attribute survives export's flatten/unflatten.
    • The numel() != 1 rejection at gguf.py:465-466 is also untested.

Code Quality

  1. The NVFP4 block decode is written twice (gguf.py:94-104 and :445-458)

    • Both places do the scale masking, the 0x7f → 0 canonicalization and the nibble split. Every other type has a single _q*_k_fields helper under "Per-type field extraction" (:161). A future fix to one copy (for example to the 0xff rule) won't reach the other.
    • Fix: factor out an _nvfp4_fields helper. With code 8 mapped to 0, the dequantize path could also use torchao's f4_unpacked_to_f32 (already used by nvfp4.py) instead of a third E2M1 table.
    • Constants in the same code:
      • The block layout is hard-coded (36, [..., :4], reshape(..., 4, 8), 16 at :469/:471), while the neighbouring k-quant constants (_Q4_K_BLOCK_BYTES, Q6_K_GROUP_SIZE) are derived.
      • QK_NVFP4 and _NVFP4_BLOCK_BYTES sit under the "integer ids" comment (:53-61).
      • _BLOCK_ELEMENTS_BY_TYPE is built by a comprehension and then mutated (:79-80), next to literal dicts.
    • Fix this →
  2. The docs still describe only k-quants decoded by the gguf package.

    • Stale places: the module docstring (gguf.py:16, :18-20, :22, and the attribution at :26-27), the _dequantize_gguf docstring (:89), the "Fused ops" comment (:140), the class docstring (:354-362), and the dequantize docstring (:419).
    • None of them mention NVFP4, to_exportable_nvfp4_tensor, or that NVFP4 decodes without the gguf package.
    • Fix this →
  3. The PR description doesn't name the AI tool. It says "Authored with AI assistance", but CLAUDE.md requires the tool name.

Recommendation

Request Changes

Items 1–3 block approval:

  • Per-tensor scale: decide what stores and applies the .scale tensor, or at least document that the caller must.
  • act_per_tensor_scale: land it with the code that uses it, or define what it means.
  • Native backend: stop it from claiming NVFP4 dequantize_gguf nodes it can't run.

The other items are small.


I couldn't run Python or the tests in this environment, so every finding comes from reading the code. All five changed files were reviewed.

  • Gather context
  • Run independent reviews
  • Consolidate findings
  • Fact-check findings
  • Post final review

…code down so A100 CI runs the NVFP4 kernels

[ghstack-poisoned]
@Gasoonjia

Copy link
Copy Markdown
Contributor Author

Thanks, addressed:

  • Native backend: _fold_gguf_dequant now raises at export for any GGUF type other than q4_k/q5_k/q6_k (this also keeps the later q8_0 out); test added in backends/native/test/test_quant.py.
  • Per-tensor scale: the module, iter_gguf, ExportableGGUFTensor and dequantize() docs now say NVFP4 values exclude the <name>.scale companion and the caller pairs it by name (the Muse Glimmer loader does). GGUF stores it as a separate tensor and iter_gguf streams tensors independently, so pairing stays with the loader.
  • to_exportable_nvfp4_tensor: keyword-only, per_tensor_scale is required (no silent 1.0), and other quant types raise NotImplementedError, matching the sibling conversions.
  • act_per_tensor_scale is documented on ExportableNVFP4Tensor: an optional static activation global scale (ModelOpt/GGUF input_scale = amax/(6*448)); None means dynamic per-row scaling; dequantize/linear ignore it and only W4A4 backends read it. A test checks it survives .to().
  • Tests: fixed the gguf pin comment; the per-tensor-scale and .to(bf16) checks now run without gguf (they compare against w.dequantize()); the gguf reference comparison covers all 256 scale bytes including 0xff (0xff is excluded only from the conversion check); added a torch.export test for both weight forms that asserts the graph node, plus the numel() != 1 and wrong-type rejections.
  • One _nvfp4_fields helper is shared by both decode paths, with named layout constants. Dequantize uses torchao's f4_unpacked_to_f32. Stale k-quant-only docs are updated, and the description names Claude Code.

@Gasoonjia

Copy link
Copy Markdown
Contributor Author

@claude /pr-review

@claude

claude Bot commented Nov 21, 2026 •

Copy link
Copy Markdown

Claude finished @Gasoonjia's task in 20m 21s —— View job


Requesting changes: an NVFP4 ExportableGGUFTensor that is missing its .scale still goes silently through fuse_along_output and linear/embedding, so nothing enforces the new rule that callers pair the scale by name. The other items are minor.

Detailed review

PR Review: #25338 (re-review)

Summary

This PR loads GGML NVFP4 blocks into ExportableGGUFTensor, decodes them in dequantize_gguf, and repacks them into ExportableNVFP4Tensor. This pass reviews the fixes pushed since the last round.

  • Blocking: the per-tensor-scale contract is documented but not enforced.
  • Minor: the input handling of act_per_tensor_scale, test gating, two code-quality points, and the PR description.

API Design

  1. An NVFP4 ExportableGGUFTensor still works as a complete weight, so nothing enforces "callers pair .scale by name" (extension/llm/export/gguf.py:28-31, :612-639; extension/llm/export/quant/convert.py:151)

    • Fusion loses the pairing.
      • fuse_along_output fuses NVFP4 GGUF shards by concatenating raw (convert.py:151), and its docstring calls the result exact (:105-107).
      • The same docstring says it runs before backend packing (:94-96). Muse's _fuse_state_dict uses it that way (examples/models/muse-glimmer/loaders/checkpoint_loader.py:354).
      • After fusion, qkv_proj.weight has no .scale to pair with, and one scalar can't hold the different scales of the shards.
      • The ExportableNVFP4Tensor entry right below (convert.py:153, :183-192) refuses a mismatched per_tensor_scale. The GGUF form fuses with no check.
    • The generic weight paths drop the scale.
      • aten.linear / embedding / t / _to_copy (gguf.py:612-639) and dequantize_weight (extension/llm/export/quant/quantize.py:294-295) compute with block-only values.
      • test_export (test_gguf_nvfp4.py:146-165) treats the scale-less dequantize_gguf(…, "nvfp4") linear as a supported export form. Yet the description says backends only consume ExportableNVFP4Tensor, and the native backend now rejects this form.
      • The torchao::dequantize_gguf docstring (gguf.py:137), which is the op backends match on, doesn't mention that the scale is missing.
    • Nothing pairs the scale in this PR.
      • to_exportable_nvfp4_tensor has no caller in the tree.
      • iter_gguf now accepts type 40, where it used to raise.
      • The Muse mmproj loader's _VISION_LAYER_RE matches only .weight / .bias (checkpoint_loader.py:541). It skips every .scale tensor (:596-598) and keeps the weight without its scale.
    • Fix: keep the pairing in the loader, but enforce it.
      • Make fuse_along_output and the NVFP4 linear / embedding / t / _to_copy paths raise for ggml_type == "nvfp4", with a pointer to to_exportable_nvfp4_tensor.
      • Add tests for those raises, plus an iter_gguf round trip that writes <name>.weight and <name>.scale.
      • Alternatively, carry the scale on the tensor. GGUFReader.tensors is a memmapped list (gguf.py:659-660), so iter_gguf can look up <name>.scale without raising peak memory.
    • Fix this →
  2. act_per_tensor_scale is documented as coming from a tensor, but it is typed as a float and stored unchanged (gguf.py:436, :440-442, :472; extension/llm/export/nvfp4.py:82, :103, :136)

    • Type:
      • The docstring says the value comes from the GGUF <name>.input_scale companion. iter_gguf yields that companion as an F32 tensor (gguf.py:672-673).
      • A caller who passes it straight through puts a tensor into tensor_attribute_names. .to(device) doesn't move it (nvfp4.py:136), and it ends up in the flatten context.
      • per_tensor_scale gets torch.as_tensor and a numel() check (gguf.py:461-465).
      • The only test passes a Python float (test_gguf_nvfp4.py:129-132).
    • Reader: the new docstring says W4A4 backends read the value. But torchao::dequantize_nvfp4 has no argument for it, so the exported graph doesn't contain it. A backend can only see it on the subclass before export, and the docstring should say so.
    • Fix: coerce it like per_tensor_scale (float(...) after a numel() == 1 check), and document where backends read it.
    • Fix this →

Testing

  1. The tests that don't need gguf still fail without it (extension/llm/export/test/test_gguf_nvfp4.py:9, :26-29)
    • import gguf runs at module scope, and requires_gguf_nvfp4 reads gguf.GGMLQuantizationType at import time.
    • Without the gguf package, the module fails to import, so none of the 8 tests that don't use gguf run. The skip only behaves as described when gguf < 0.19 is installed, which contradicts the reply's claim that these tests "now run without gguf".
    • Fix: guard the import with _HAS_GGUF like the sibling test_gguf.py:31-36, and gate requires_gguf_nvfp4 on it.
    • Fix this →

Code Quality

  1. _BLOCK_ELEMENTS_BY_TYPE gives every type 256 elements by default (gguf.py:91-92)

    • The comprehension sets QK_K for every key of _BLOCK_BYTES_BY_TYPE and then overrides nvfp4.
    • A type added only to _BLOCK_BYTES_BY_TYPE would silently get a K 8× too large in __new__ (:392) and in the fake kernel (:143-145). The q8_0 planned later in the stack (32 elements in 34 bytes) would hit this.
    • Fix: use a literal dict like the two above it, so a missing type raises KeyError.
  2. The 0xff rule is still split across the callers of _nvfp4_fields (gguf.py:155-157, :164)

    • The docstring says the scale bytes come back canonicalized to float8_e4m3fn. In fact raw 0xff comes back as 0x7f, which is NaN in E4M3fn.
    • Each caller patches this on its own: _dequantize_gguf substitutes 480 (:111-112), and to_exportable_nvfp4_tensor rejects the byte (:457-460). That is the kind of duplicated rule the shared helper was meant to remove.
    • Fix: handle 0xff inside the helper, or document there that callers must handle it.
    • Fix this → (covers items 4 and 5)
  3. The PR description is stale.

    • It still ends with "Authored with AI assistance." and names no tool. CLAUDE.md requires the tool name, and the reply says the description now names Claude Code.
    • It still calls the per-tensor scale "optional", but per_tensor_scale is now a required argument (gguf.py:434).

Recommendation

Request Changes

Item 1 blocks approval. Today an NVFP4 ExportableGGUFTensor can be fused, exported or run without its .scale, and nothing raises. Raising in those paths keeps the loader-side pairing and closes the gap. Items 2–6 are minor.


Python couldn't be run in this environment, so all findings come from reading the code. The GGML bit layout was checked by hand, not against gguf-py or llama.cpp. All seven changed files were reviewed.

  • Gather context
  • Run independent reviews
  • Consolidate findings
  • Fact-check findings
  • Post final review

@Gasoonjia

Copy link
Copy Markdown
Contributor Author

Round 2:

  • Pairing is now enforced. An NVFP4 ExportableGGUFTensor is not a usable weight on its own: dequantize(), linear/embedding/t/_to_copy raise ValueError and fuse_along_output raises TypeError, all pointing to to_exportable_nvfp4_tensor(per_tensor_scale=...). The explicit block decode stays available as torchao::dequantize_gguf(raw, "nvfp4"), documented as excluding the scale. Tests cover each raise, plus an iter_gguf round trip that writes <name>.weight and <name>.scale and pairs them.
  • act_per_tensor_scale accepts a float or a one-element tensor, is numel-checked, and is stored as a Python float. The docstring now says it never reaches the exported graph: W4A4 backends read it from the tensor when packing.
  • gguf is imported behind a guard, so the 9 gguf-independent tests run without the package (checked with gguf blocked).
  • _BLOCK_ELEMENTS_BY_TYPE is a literal dict; the later q8_0 commit adds its entry. The 0xff rule now lives only in _nvfp4_fields, which returns the E4M3 bytes and the GGML-decoded float scales; conversion rejects any scale whose two forms differ.
  • The native backend test now reaches _fold_gguf_dequant through a direct dequantize_gguf call, because the weight path raises earlier.
  • The commit summary names Claude Code and says the scale is required.

@Gasoonjia

Copy link
Copy Markdown
Contributor Author

@claude /pr-review

@claude

claude Bot commented Nov 21, 2026 •

Copy link
Copy Markdown

Claude finished @Gasoonjia's task in 19m 30s —— View job


Requesting changes for one line: the new test_gguf_nvfp4 Buck target is missing its :quant dependency, so it can't import. Two minor points remain: the per-tensor scale gets silently rounded on the standard loader path, and the PR body is stale.

Detailed review

PR Review: #25338 (round 3)

Summary

This PR loads GGML NVFP4 blocks as ExportableGGUFTensor, decodes them in torchao::dequantize_gguf, and repacks them losslessly into ExportableNVFP4Tensor. A missing Buck dependency stops the new test target from importing. Two smaller issues remain.

Infrastructure

  1. The test_gguf_nvfp4 Buck target can't import its own module (extension/llm/export/test/BUCK:42-52, extension/llm/export/test/test_gguf_nvfp4.py:12)
    • The test imports fuse_along_output from executorch.extension.llm.export.quant.convert. That module is only in the //executorch/extension/llm/export:quant target (extension/llm/export/BUCK:155-163).
    • The same target also holds the package __init__.py, which fuse_along_output imports from (extension/llm/export/quant/convert.py:137-142).
    • The new target lists only :gguf and :nvfp4, and neither depends on :quant.
    • Under Buck the module fails with ModuleNotFoundError, so none of its 12 tests run. That includes the 9 tests the round-2 reply says run without gguf. OSS pytest is not affected.
    • Fix: add "//executorch/extension/llm/export:quant" to the deps.
    • Fix this →

API Design

  1. The standard GGUF loader rounds the .scale companion before callers can pair it, and the conversion accepts the rounded value without complaint (extension/llm/export/gguf.py:458-461, :482-489, :680-681)
    • The cast: the new contract says callers pair <name>.scale by name. But iter_checkpoint runs maybe_cast(convert(fqn, value), dtype) on every tensor (extension/llm/export/load.py:265), and maybe_cast casts plain tensors to dtype (quant/convert.py:61-64).
      • With iter_checkpoint(path, dtype=torch.bfloat16), the F32 scale reaches the caller already rounded to bf16, with up to 2^-8 (about 0.4%) relative error.
      • The convert callback sees one tensor at a time, so it can't pair the scale before the cast.
      • The Muse Glimmer GGUF path loads this way: _atomic_sd_from_gguf passes dtype=activation_dtype (examples/models/muse-glimmer/loaders/checkpoint_loader.py:373-384).
    • The silent upcast: to_exportable_nvfp4_tensor turns the bf16 tensor back into float32 with torch.as_tensor(..., dtype=torch.float32) and raises nothing.
      • Every element of the weight is then off by the same factor, so the "losslessly" claim doesn't hold.
      • act_per_tensor_scale takes the same path (:488, :498).
    • Scope: no code in this PR calls the conversion this way. Whether diff 3 hits it depends on how the Muse loader pairs the scale.
    • Fix: reject tensor scales whose dtype is not float32. GGUF stores .scale as F32, so any other dtype means the value was already cast. At minimum, document on iter_gguf and to_exportable_nvfp4_tensor that the scale has to be paired before any dtype cast.
    • Fix this →

Code Quality

  1. The PR body is still stale.
    • In the copy fetched for this run, it still says "optional per-tensor scale from the GGUF .scale companion", but per_tensor_scale is now required (gguf.py:452).
    • It still ends with "Authored with AI assistance." and names no tool, which CLAUDE.md requires.
    • The round-2 reply says the commit summary was updated, but the change hasn't reached the PR body. The head-branch commits are ghstack-poisoned, so the summary couldn't be checked from here.

Recommendation

Request Changes

Only item 1 blocks, and it's a one-line BUCK fix. Items 2 and 3 are minor.


Python couldn't be run in this environment, so every finding comes from reading the code. All eight changed files were reviewed.

  • Gather context
  • Run independent reviews
  • Consolidate findings
  • Fact-check findings
  • Post final review

@Gasoonjia

Copy link
Copy Markdown
Contributor Author

Round 3:

  • test_gguf_nvfp4 now depends on //executorch/extension/llm/export:quant.
  • Scale rounding: to_exportable_nvfp4_tensor now rejects tensor scales that are not float32 (GGUF stores both companions as F32, so another dtype means a loader cast already rounded them). Python floats are still accepted. The docstrings on it and on iter_gguf say to pair <name>.scale before any dtype cast. A test covers bf16 per_tensor_scale and act_per_tensor_scale.
  • The Muse Glimmer loader is not affected: it reads both companions with GGUFReader as F32, not through iter_checkpoint.
  • The commit summary is current: the scale is required and the tool is Claude Code. It will be copied to the PR body.
  • The gguf>=0.19 pin goes in requirements-dev.txt through a separate OSS-only PR, Require gguf>=0.19 for the GGUF tests #25420, because .github/ and requirements are OSS-only.

@Gasoonjia

Copy link
Copy Markdown
Contributor Author

@claude /pr-review

@claude

claude Bot commented Nov 21, 2026 •

Copy link
Copy Markdown

Claude finished @Gasoonjia's task in 24m 46s —— View job


Approve with three minor follow-ups: the generic GGUF loader now loads unpaired NVFP4 weights that only fail later at forward or export, ExportableNVFP4Tensor doesn't enforce the documented type of act_per_tensor_scale, and without gguf ≥ 0.19 no test checks the row and block order of _nvfp4_fields or which code bytes each K16 group reads.

Detailed review

PR Review: #25338 (round 4)

Summary

This PR loads GGML NVFP4 blocks as ExportableGGUFTensor, decodes them in torchao::dequantize_gguf, and repacks them into ExportableNVFP4Tensor once the caller pairs the .scale companion. Nothing blocks this round. Three minor items remain: one loader gap, one unenforced attribute contract, and one test gap.

API Design

  1. The generic GGUF loader can't do the pairing, so unpaired NVFP4 weights now get past load (extension/llm/export/gguf.py:699-706; extension/llm/export/load.py:198-208, :261-267, :292)

    • Before: iter_gguf raised Unsupported GGUF quant type for type 40, so an NVFP4 file failed at load.
    • Now: iter_checkpoint / load_checkpoint yield each NVFP4 weight as an unpaired ExportableGGUFTensor. These loaders can't do the pairing that callers are told to do:
      • convert sees one tensor at a time.
      • When dtype= is set, maybe_cast casts <name>.scale before anything can pair it, and the new float32 check (gguf.py:484-489) then rejects that scale. muse-glimmer's _atomic_sd_from_gguf sets it to the activation dtype.
    • Effect: load_checkpoint assigns the weight as if it were "already in its final portable form" (load.py:292). The failure moves to the first forward, export or fuse_along_output. The error message tells the user to call to_exportable_nvfp4_tensor(per_tensor_scale=...), which none of the loader's callbacks can do.
    • Fix: pairing is the loader's job, and _iter_gguf is the generic loader. Either:
      • pair <name>.scale / <name>.input_scale there before the cast, or
      • have iter_checkpoint reject an NVFP4 ExportableGGUFTensor, so the error stays at load time.
    • Fix this →
  2. ExportableNVFP4Tensor doesn't enforce the documented type of act_per_tensor_scale (extension/llm/export/nvfp4.py:75-76, :84, :105, :138; gguf.py:493-506)

    • The gap: the class docstring says the attribute is a Python float or None. The numel check and the float() call exist only in ExportableGGUFTensor.to_exportable_nvfp4_tensor, and __new__ stores whatever value it gets.
    • Who hits it: other code builds the class directly (nvfp4.py:221, examples/models/muse-glimmer/loaders/checkpoint_loader.py:264). Suppose one of these passes a checkpoint's input_scale tensor:
      • The tensor goes into the __tensor_flatten__ metadata, because the attribute is listed in tensor_attribute_names.
      • .to(device) doesn't move it.
    • Fix: do the coercion in __new__, and keep only the GGUF float32 check in the conversion.
    • Fix this →

Testing

  1. Without gguf ≥ 0.19, no test checks the row and block order of _nvfp4_fields, or which code bytes each K16 group reads (extension/llm/export/test/test_gguf_nvfp4.py:32-39, :66-83, :116-119, :246-255; extension/llm/export/test/BUCK)
    • The gated tests may not run: every comparison against the gguf decoder is behind requires_gguf_nvfp4, which skips silently. Require gguf>=0.19 for the GGUF tests #25420 pins the gguf version only for OSS. The Buck target uses fbsource//third-party/pypi/gguf, and I couldn't check which version that resolves to.
    • The tests that always run don't cover ordering:
      • test_hand_built_block and test_unrepresentable_scale_is_rejected build their expected values by hand, but each uses one row and one block. Every group gets the same code bytes (.repeat(4) at :73, 0x11 at :249). So only differences between the group scales could expose a group mix-up.
      • test_per_tensor_scale_and_dtype uses 7 rows of 3 blocks, but its expected values come from blocks(w), which calls the same _nvfp4_fields. test_export checks only that the paths agree with each other, and test_fake_shape checks only shapes.
    • Result: if a change to _nvfp4_fields swapped rows or blocks, or gave a group another group's code bytes, every test that runs without gguf would still pass.
    • Fix: extend a hand-built test to at least two rows and two blocks, give each group different codes, and build the expected values from the test's own E2M1 table.
    • Fix this →

Recommendation

Approve

Nothing blocks. Item 1 matters most before landing: it decides whether an NVFP4 GGUF fails at load or only at export, for every caller of the generic loader. Items 2 and 3 are small follow-ups.


I couldn't run Python here, so every finding comes from reading the code. All eight changed files were reviewed. I didn't check the GGML NVFP4 bit layout against llama.cpp or gguf-py source.

  • Gather context
  • Run independent reviews
  • Consolidate findings
  • Fact-check findings
  • Post final review

This branch was previously deployed

1 inactive deployment
cadence — ca5ea65f Deployed Nov 21, 2026 by Gasoonjia via hifi-op-test / hifi4 #39974
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. meta-exported

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant